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ABSTRACT 

THE QUALITY OF RESEARCH ON EDUCATION ; 

AN EMPIRICAL STUDY OF RESEARCHERS AND THEIR WORK 

Caroline Hodges Persell 



The problem of this study is one of measuring and explaining the 
quality of recent research on education. The problem was approached by 
locating ail empirical research on education published in 113 American 
behavioral science and education journals in 1967-68 (833 articles; plus 
all empirical research papers presented at the 1968 annual meeting of 
the American Educational Research Association Can additional 267 papers). 
Three types of data were collected on this sample of 1,100 papers and 

their authors : 

(1) All 1,100 research papers were qualitatively analyzed with 

respect to their substance and methodology. 

(2) A stratified random sample of 390 papers was evaluated by a 
national panel of 39 judges with respect to three dimensions of quality- 
contribution to theory, contribution to educational practice, and use of 
research methods , A five point scale, developed and validated in earlier 
research by the author, was used to rate each of the three dimensions 

of quality. A sample of papers was rated in common by five judges each, 
to assess agreement among the judges. Consensus was measured by the 
average deviation from the mean, and the average of the average deviations 
was .62 . Papers eliciting the greatest and least agreement among 
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were qualitatively analyzed in an effort to ascertain what is related 
to greater or lesser consensus in evaluation. Characteristics of judges 
were also analyzed to see if they were related to variation in agree- 
ment . 

(3) A twelve page written questionnaire was mailed to all 1 ,100 
authors in the sample, and 83 per cent returned completed questionnaires. 
The questionnaire collected data on the authors ' background, training, 
career lines, attitudes, research contexts, and topic selection. 

These three types of data were analyzed using multivariate 
tabular analysis. Throughout the analysis, authors trained in sciwois 
of education were compared with those trained in the behavioral sciences . 

Professional socialization (as measured by type of doctorate, 
practice-oriented socialization, graduate research experience, career 
research experience, and research orientation) was found to be strongly 
related to research quality, and to have a stronger relationship to quality 
than undergraduate academic performance. Behavioral scientists had more 
research-’ related socialization and did better research than education 
doctorates . 

The positive relationship between research socialization and 
quality was contingent upon, or intensified by, certain background 
characteristics. In particular, females and those who classified them- 
selves as either areligious or Jewish were more likely than males or 

those belonging to other religious groups to do better research when 
they had more research socialization-. This finding was partially 
I explained by the greater intellectual orientation of the females, Jews, 
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and areligious researchers in the sample studied. In addition, younger 
researchers were generally more likely than older ones to do better 
research, in part because of their greater research socialization. 

Other experiences besides research socialization were related to 
better quality, including institutional setting (working in a university 
or in a private research agency rather than in a college, school system, 
or state department of education); and being a staff member of a 
university research center „ 

In addition to the foregoing analyses, the study explores the 
relationship between research quality and subsequent rewards, specifically 
the rewards of being promoted in academic rank, moving from a- less 
prestigious to a more prestigious department, and applying for and 
receiving a research grant . The data suggest that in the behavioral 
sciences authors of better research papers are more likely to be rewarded 
than authors of average or poor papers , whereas among those trained in 
education, authors of better papers are no more likely to be rewarded 
than authors of mediocre or poor papers. Several explanations of this 

finding were considered. 
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CHAPTER I 
INTRODUCTION 1 

In this volume we are concerned with the problem of what 
affects performance of the research role in education , or more 
specifically, what processes explain variation in role performance. 

Our concern differs in at least two wavs from that of earlier work 
which has touched upon this problem. First, we will assess role per- 
formance in terms of quality rather than quantity. Second, we view 
the processes affecting role performance as interdependent aspects of 
a social system. Thus we are interested in how such characteristics 
of the social system of research as recruitment, role socialization, 
career lines, orientations, organizational contexts, and the reward 
and communication sub -systems interact to produce research of varying 
quality , 

We will study performance of the research role in education. 
Education presents a strategic research site for studying variation in 
the quality of research inasmuch as researchers in education are 
drawn from a wide array of academic fields , thereby giving us the 
unique oppoTtimxty to study behavioral science research in a variety of 



1 This research was supported in large part by a U.^, Office of 
Education Contract; This support is gratefully acknowledged. 
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disciplines. Further, the ‘'conjunctive domain" 1 of education, as its 
relation to the disciplines has been characterized, provides an 
opportunity to compare research in the context of a professional school 
with that in traditional academic disciplines, thus offering a chance 
to explore possible constraints imposed upon research in a professional 
setting , 

In addition to serving as a site for theoretically oriented 
research, education is an area of great social concern, American educa- 
tion faces severe problems . There are the problems of educating the 
disadvantaged and the gifted, the difficulties of preparing children 
for ever increasing technological change, and the problems of coping 
with conflicting values and interests among different segments of 
society. Although these and other grave problems need attention, edu- 
cational systems are so busy with their daily tasks that they are 
unable to generate the new understanding and techniques needed to solve 
such problems. Therefore, educational practitioners must look to 
researchers for the insight Mid ideas to overcome their problems and 
meet society*s expectations. In this framework, the problem of how 
research on education can be improved looms large. In particular, it 
is assumed that if educational research is improved it will eventually 
benefit practice. The better the research, the more acute our 

1 Norman IV, Storer, "The Organization and Differentiation of the 
Scientific Community’: Basic Disciplines, Applied Research and Con- 

junctive Domains," prepared for the Colloquium on "Improving the Social 
and Communication Mechanisms of Educational Research sponsored by 
the American Educational Research Association, Washington , D, C . , 
November 21-22, 1968 (xerox) . 
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understanding of educational processes, and eventually one hopes the 
greater the chances of solving educational problems . 

Our approach is first to discuss and measure the quality of 
research on education, and then to analyze the processes that affect 
it. Following a description of our sample in Chapter 1 , the volume is 
divided into two major parts. Part One, comprising Chapters III 
through V, addresses the problem of describing and measuring research 
quality. Part TWo, comprising Chapters VI through IX, attempts to 
identify the major variables that are related to quality, to formulate 
a model of how these variables might be interrelated and finally to 
test the hypothesized interrelationships . Because our data reveal 
complex interrelationships , we have presented the model as it develops 
in a series of flow charts, one or more of which appear in each of the 
chapters of Part Two . 

As just noted. Chapter II is primarily descriptive of the 
papers mid authors in the sample studied, and of our methods of data 
collection. Where did the papers appear? What were the primary sub- 
stantive topics of the papers and what research strategies were used? 

Is there any relation between the research methods and the types of 
problems that were studied? What is the background, training, and 
organizational contexts of these authors of recent research on 
education? These are tne types of questions that will be considered. 

Chapter III, the first chapter in Part One, deals with the 
nature of research quality and the problems of measuring it. We review 
several studies that have measured research outcomes, and try to 

eric: . : 
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specify the assumptions underlying the various methods , We consider 
whether some methods are more appropriate than others under certain 
circumstances, and present the rationale for the method used in this 
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study. 



In Chapter IV we discuss the rating form used to measure 
quality in this study and describe how it was administered. Then we 
examine the extent and nature of agreement and disagreement among the 
judges on commonly- rated papers, and try to ascertain whether certain 
characteristics of judges or papers contribute to greater or lesser 
consensus ■ In particular, we do a qualitative analysis of the papers 
which elicited the greatest and the least agreement among raters in an 
effort to understand why some research reports are rated similarly and 
others are not. Also, we examine judges* specialties, locations, 
training, age, and sex to see if the characteristics of judges explain 
agreement. Then we focus on each dimension of quality (theory, prac- 



tice, and research methods) to see if one dimension elicits more con- 
sensus among judges than another. As a way of learning more about what 



affects agreement among judges on each dimension, we also qualitatively 
analyze the papers with the most and the least agreement among judges 
on each of the three dimensions . In short, the purpose of the analysis 



is to see if we can identify general characteristics of research papers 
or judges that are related to consensu^ among evaluators . 

We present the results of the judges * ratings in Chapter V, 
showing the distribution of research quality in this sample of authors . 
Is research quality distributed normally or is it skewed toward one end 

iJJI 
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of the rating scale? Then, using qualitative comments by the judges, 
we summarize the weaknesses in papers rated poor on substance, research 
design,,, data collection, or data analysis. In this chapter we also 
examine the interrelation of the three dimensions of quality, and 
present qualitative analyses of those papers that were rated high on 
one dimension and low on another. These analyses were done to provide 
insights into both the rating process and the nature of good research. 

Part TWo of this volume includes Chapters VI to IX, There we 
shift our focus from the phenomenon to be explained, that is, research 
quality, to a consideration of the processes that affect it. 

One of the key processes that we believe affects research 
quality is role socialization , and we devote Chapter VI to a discussion 
of tills process , Earlier empirical studies of research training have 
found several experiences which seem to have positive or negative 
consequences for research involvement or productivity. These studies, 
together with more general literature on socialization for professional 
roles, enable us to identify five elements of role socialization that 
we consider central for explaining variation in quality. We discuss 
each of these elements and then develop a model of the interrelation- 
ships : among them and quality, in the second half of the chapter we 
analyze the m °del and try to specify 

;tiie processes underlying these associations . The chapter on role 
socialization ; precedes the one on background because we considered it 
necessary to examine, the elements of this complex, phenomenon before we 
cotiid explore ^e pfi^ in affecting the relationship 

#een role socialization and quality. 
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In Chapter VII we turn to the question of selective recruit- 
ment, There we discuss the background of authors of educational 
research, including their sex, religion, socio-economic status, age, 
and undergraduate academic performance. Traditionally, educational 
research has been done by upwardly mobile, older, male Protestants, 

We consider the evidence and opinions about how these background 
characteristics may be associated with quality . We also explore 
whether background characteristics affect the relationship between 
role socialization and quality, and if so, why. 

In Chapter VIII we turn from the authors* recruitment and role 
socialization to their work settings . Observers of educational 
research often mention structural settings as key factors in explain- 
ing variation in research quality. We consider several frequently 
mentioned settings, including: organizational affiliation, i.e,, 

university, college, specialized agency, school system or other set- 
tings departmental affiliation j departmental prestige; and university 

research center affiliation. Where relevant, we examine the inter- 

* 

relation of work settings, background, role socialization, and quality, 
in an effort to specify the conditions under which work settings are 
related to better research* 

We move beyond the authors 1 immediate work context in Chapter 
IX to the larger social system in which he pursues his career. Here 
we seek to describe the reward and communication systems and to analyse 
how their operation may affect quality. Does the reward system 

encourage better or poorer research? What are some of the other 

O 
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consequences of the way it operates? Why does the reward system 
operate the way it does in this social system? These are some of the 

questions we consider in Chapter IX. 

In Chapter X we summarize the findings of our study, and 
conclude by presenting a set of conditions that seem to enhance the 
quality of empirical research on education. 



CHAPTER II 



THE NATURE OF THE POPULATION: A DESCRIPTION 

OF THE PAPERS AND RESEARCHERS 



Nature of the Population 

The concern of this study is recent empirical research on 
education, as conducted by behavioral scientists and by people in 
education. It is important to focus upon discrete research papers in 
order to assess their quality. Therefore, we selected a time sample 
of recent authors of research papers whose articles were printed in 
education and behavioral science journals published in the United 
States during the period July 1967 through Jvme 1968, or whose papers 
were presented at the 1968 American Educational Research Association 
(AERA) annual meeting. 

The educational and behavioral science journals were selected 
as follows; First, we checked a number of bibliographic references 



hie drew the education journals from the following sources:^ 
America's Education Press, a classified list of periodicals issued in 
the U.2. and Canada, The Educational Press Association of America, 
Yearbook 29. 1966 i Bibliographic Index, a cumulative bibliography of 
bibliographies (New York: H, W. Wilson Co., 1967 and 1968); M. Blaug, 

Economics of Education : A Selected Annotated Bibliography (New York 

and Oxford: Pergamon Press, 1966) ; Education Index (New York : II . W . 

Wilson Co., Vol. 40, No. 1, 4); Saul Hemer, Janet D, Griffith, and 
Mary Hemer, Study of Periodicals a nd Serials in Education, Final ^ 
Report, Project No, BR 7-9003, U.S. Office of Education, Department of 
Health, Education and Welfare (Washington, D.C.: Hemer and Company , 

June 28, 1968) , ERIC document number ED 017 747; and Psychological 
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to develop an inclusive list of journals published in the United States 
which might contain empirical research articles on education, W© 
deliberately included as many discipline journals as possible so we 
could compare education doctorates with behavioral scientists . 

Second, we excluded several classes of publication prior to 
scanning. These include: review journals which merely summarize 

existing research; publications of state teachers' associations, state 
departments of education, or proceedings of professional associations ; 
dissertation abstracts; yearbooks; monographs; journals with a circula- 
tion of less than 1,000; 1 privately commissioned and circulated papers 
and reports (such as those done by the RAND Corporation) if they are 



Abstracts (Washington, D.C.: American Psychological Association, 1967). 

We drew the behavioral science journals from the following 
references: Bibliographic Index, op. cit . ; Economic Abstracts, V. IV- 1, 

No. 21, February 1956 (New York: New York University, 1956); Index of 

Economic Journals, Vol. VII, 1964-1965, prepared under the auspices of 
the American Economic Association (Homewood, 111.: Richard D. Irwin, 

Inc,, 1967); International Bibliography of the Social Sciences : 
Economics, 1966, Vol. XV (Chicago: Aldine; London: Tavistock, 1968); 

I nternational Bibliography of the Social Sciences: Political Science, 

Vol, XV (Chicago: Aldine; London: Tavistock, 1966); International 

Bibliography of the Social Sciences: So cial and Cultural Anthropology, , 

Vol . XII (Chicago : Aldine ; London : Tavistock , prepared under the 

auspices of the International Union of Anthropological and Ethnological 
Sciences by ICSSD, with the support of UNESCO, 1967) ; International 
Bibliography of the Social Sciences: Sociology (Chicago: Aldine; 

London: Tavistock, 1966); International Political Science Abstracts , 

Vol. XVII, No. 1, October- December 1966 (Oxford: Basil Blackwell, 

1967) ; Psychological Abstracts, op . cit . ; and Sociological Abstracts^ 
(Hew York: International Sociological Association, Eastern Sociological 

Society, and Midwest Sociological Society, Vol. 13, Nos, 1-7, covering 
1965, c, 1964-67). 

Hfe determined circulation of less than 1,000 by consulting the 
following: America's Education Press, a classified list of periodicals 

issued in the U,S. and Canada, Yearbook 29 (Educational Press Associa- 
tion of America, 1966) ; and Directory: Newspapers and Periodicals 1968 

(Philadelphia, Pa,: N. W, Ayer said Sons, 1968) , """ 
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not reported in journal articles; and journals dealing with physical 
education, home economics, industrial arts, nursing, or school safe y. 

A list of 505 journals emerged from the above selection 
process. Of these, we eliminated 131 without scanning because they 
were beyond the defined scope of the population, they were impossible 
to locate, or for some other reasons such as not publishing empirical 
research, or being a regional, review, or foreign publication. (See 
Appendix II -1.) We scanned the remaining 372 journals for the desig 
nated year. Of these journals 259 did not yield any empirical research 
articles about education (Appendix II-2 contains the names and reasons) . 
The balance, or 113 journals (Appendix 11-3), yielded a total of 846 

empirical research articles on education P 

In addition to the 846 journal articles, the sample includes 
269 empirical research papers on education presented at the 1968 annual 
American Educational Research Association (AERA) meeting. The people 
in our sample, then, are recent authors of empirical research papers on 
education, which was either published in a journal or presented at the 
1968 AERA meeting. We included only those authors with addresses in 
the United States because the study was confined to American educa- 
tional research. 2 If an author had published more than one paper 

l A samples of articles was scanned independently by three oilier 
neonle from several fields, as a way of verifying our decisions about 
whidh articles should be included in the population. agreed inal 

but six cases which were borderline topics dealing with fairly basic 

L. psychology. After considering the situation, we . 

decided to include these borderline cases . 

2 ln addition to the six articles mentioned in notel, we sub- 
sequently discovered twelve people, such as foreign students, who had 
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